{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2d97fef3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 9.28 ms (started: 2021-08-13 02:19:30 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 自动计算cell的计算时间\n",
    "%load_ext autotime\n",
    "\n",
    "%config InlineBackend.figure_format='svg' #矢量图设置，让绘图更清晰"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3ea088e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "origin\tgit@github.com:ustchope/kerastuner.git (fetch)\n",
      "origin\tgit@github.com:ustchope/kerastuner.git (push)\n",
      "[master 2610353] 更新 #2 Aug 13, 2021\n",
      " 2 files changed, 328 insertions(+), 1 deletion(-)\n",
      " create mode 100644 \"\\347\\274\\226\\345\\206\\231\\350\\207\\252\\345\\267\\261\\347\\232\\204 Tuner \\344\\273\\245\\346\\224\\257\\346\\214\\201\\350\\207\\252\\345\\256\\232\\344\\271\\211\\350\\256\\255\\347\\273\\203\\345\\276\\252\\347\\216\\257.ipynb\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To github.com:ustchope/kerastuner.git\n",
      "   866c520..2610353  master -> master\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 4.17 s (started: 2021-08-13 02:21:28 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# 增加更新\n",
    "git add *.ipynb\n",
    "\n",
    "git remote -v\n",
    "\n",
    "git commit -m '更新 #2 Aug 13, 2021'\n",
    "\n",
    "git push origin master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5865aa77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.55 s (started: 2021-08-13 02:19:45 +08:00)\n"
     ]
    }
   ],
   "source": [
    "#设置使用的gpu\n",
    "import tensorflow as tf\n",
    "\n",
    "gpus = tf.config.list_physical_devices(\"GPU\")\n",
    "\n",
    "if gpus:\n",
    "   \n",
    "    gpu0 = gpus[0] #如果有多个GPU，仅使用第0个GPU\n",
    "    tf.config.experimental.set_memory_growth(gpu0, True) #设置GPU显存用量按需使用\n",
    "    # 或者也可以设置GPU显存为固定使用量(例如：4G)\n",
    "    #tf.config.experimental.set_virtual_device_configuration(gpu0,\n",
    "    #    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]) \n",
    "    tf.config.set_visible_devices([gpu0],\"GPU\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb4ad5f9",
   "metadata": {},
   "source": [
    "# 介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694cb69b",
   "metadata": {},
   "source": [
    "KerasTuner 可以轻松执行分布式超参数搜索。 从在本地运行单线程扩展到在数十或数百个工作线程上并行运行，无需更改您的代码。 分布式 KerasTuner 使用首席工人模型。 负责人运行一个服务，工作人员向该服务报告结果并查询接下来要尝试的超参数。 主管应该在单线程 CPU 实例上运行（或者作为一个单独的进程运行在其中一个工作线程上）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13128b75",
   "metadata": {},
   "source": [
    "# 配置分布式模式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "787dff28",
   "metadata": {},
   "source": [
    "为 KerasTuner 配置分布式模式只需要设置三个环境变量："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3206f821",
   "metadata": {},
   "source": [
    "`KERASTUNER_TUNER_ID`：对于主要进程，这应该设置为“chief”。 其他工人应该被传递一个唯一的 ID（按照惯例，“tuner0”、“tuner1”等）。\n",
    "\n",
    "`KERASTUNER_ORACLE_IP`：主要服务应该运行的 IP 地址或主机名。 所有工作人员都应该能够解析和访问此地址。\n",
    "\n",
    "`KERASTUNER_ORACLE_PORT`：主要服务应该运行的端口。 这可以自由选择，但必须是其他工人可以访问的端口。 实例通过 gRPC 协议进行通信。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce9f62f4",
   "metadata": {},
   "source": [
    "相同的代码可以在所有工人上运行。 分布式模式的其他注意事项是：\n",
    "* 所有工作人员都应该可以访问一个集中式文件系统，他们可以将结果写入其中。\n",
    "* 所有工作人员都应该能够访问调优所需的必要培训和验证数据。\n",
    "* 为了支持容错，`overwrite`应该在 `Tuner.__init__` 中保持为 `False` （`False` 是默认值）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e70c1bd",
   "metadata": {},
   "source": [
    "首席服务的示例 bash 脚本（页面底部的 run_tuning.py 示例代码）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71538b60",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "export KERASTUNER_TUNER_ID=\"chief\"\n",
    "export KERASTUNER_ORACLE_IP=\"127.0.0.1\"\n",
    "export KERASTUNER_ORACLE_PORT=\"8000\"\n",
    "python run_tuning.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5f09fb6",
   "metadata": {},
   "source": [
    "工人的示例 bash 脚本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c9f0c37",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "export KERASTUNER_TUNER_ID=\"tuner0\"\n",
    "export KERASTUNER_ORACLE_IP=\"127.0.0.1\"\n",
    "export KERASTUNER_ORACLE_PORT=\"8000\"\n",
    "python run_tuning.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82005158",
   "metadata": {},
   "source": [
    "# tf.distribute 的数据并行性"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9204f500",
   "metadata": {},
   "source": [
    "KerasTuner 还通过 tf.distribute 支持数据并行。 数据并行和分布式调优可以结合起来。 例如，如果您有 10 个 worker，每个 worker 上有 4 个 GPU，您可以使用 tf.distribute.MirroredStrategy 在 4 个 GPU 上运行 10 次并行试验，每次试验训练。 您还可以通过 tf.distribute.experimental.TPUStrategy 在 TPU 上运行每个试验。 目前不支持 tf.distribute.MultiWorkerMirroredStrategy，但对此的支持在路线图上。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b00cf48",
   "metadata": {},
   "source": [
    "# 示例代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46cf0203",
   "metadata": {},
   "source": [
    "设置上述环境变量后，下面的示例将运行分布式调整并通过 tf.distribute 在每个试验中使用数据并行性。 该示例从 tensorflow_datasets 加载 MNIST 并使用 Hyperband 进行超参数搜索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8795f55d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial 2 Complete [00h 00m 11s]\n",
      "val_accuracy: 0.18000000715255737\n",
      "\n",
      "Best val_accuracy So Far: 0.25\n",
      "Total elapsed time: 00h 00m 27s\n",
      "INFO:tensorflow:Oracle triggered exit\n",
      "time: 30.2 s (started: 2021-08-13 02:19:51 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import keras_tuner as kt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def build_model(hp):\n",
    "    \"\"\"Builds a convolutional model.\"\"\"\n",
    "    inputs = tf.keras.Input(shape=(28, 28, 1))\n",
    "    x = inputs\n",
    "    for i in range(hp.Int(\"conv_layers\", 1, 3, default=3)):\n",
    "        x = tf.keras.layers.Conv2D(\n",
    "            filters=hp.Int(\"filters_\" + str(i), 4, 32, step=4, default=8),\n",
    "            kernel_size=hp.Int(\"kernel_size_\" + str(i), 3, 5),\n",
    "            activation=\"relu\",\n",
    "            padding=\"same\",\n",
    "        )(x)\n",
    "\n",
    "        if hp.Choice(\"pooling\" + str(i), [\"max\", \"avg\"]) == \"max\":\n",
    "            x = tf.keras.layers.MaxPooling2D()(x)\n",
    "        else:\n",
    "            x = tf.keras.layers.AveragePooling2D()(x)\n",
    "\n",
    "        x = tf.keras.layers.BatchNormalization()(x)\n",
    "        x = tf.keras.layers.ReLU()(x)\n",
    "\n",
    "    if hp.Choice(\"global_pooling\", [\"max\", \"avg\"]) == \"max\":\n",
    "        x = tf.keras.layers.GlobalMaxPooling2D()(x)\n",
    "    else:\n",
    "        x = tf.keras.layers.GlobalAveragePooling2D()(x)\n",
    "    outputs = tf.keras.layers.Dense(10, activation=\"softmax\")(x)\n",
    "\n",
    "    model = tf.keras.Model(inputs, outputs)\n",
    "\n",
    "    optimizer = hp.Choice(\"optimizer\", [\"adam\", \"sgd\"])\n",
    "    model.compile(\n",
    "        optimizer, loss=\"sparse_categorical_crossentropy\", metrics=[\"accuracy\"]\n",
    "    )\n",
    "    return model\n",
    "\n",
    "\n",
    "tuner = kt.Hyperband(\n",
    "    hypermodel=build_model,\n",
    "    objective=\"val_accuracy\",\n",
    "    max_epochs=2,\n",
    "    factor=3,\n",
    "    hyperband_iterations=1,\n",
    "    distribution_strategy=tf.distribute.MirroredStrategy(),\n",
    "    directory=\"results_dir\",\n",
    "    project_name=\"mnist\",\n",
    "    overwrite=True,\n",
    ")\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# Reshape the images to have the channel dimension.\n",
    "x_train = (x_train.reshape(x_train.shape + (1,)) / 255.0)[:1000]\n",
    "y_train = y_train.astype(np.int64)[:1000]\n",
    "x_test = (x_test.reshape(x_test.shape + (1,)) / 255.0)[:100]\n",
    "y_test = y_test.astype(np.int64)[:100]\n",
    "\n",
    "tuner.search(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    steps_per_epoch=600,\n",
    "    validation_data=(x_test, y_test),\n",
    "    validation_steps=100,\n",
    "    callbacks=[tf.keras.callbacks.EarlyStopping(\"val_accuracy\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2517d0c2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tf2keras]",
   "language": "python",
   "name": "conda-env-tf2keras-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
